Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.

This paper introduces a probabilistic model for unconditional word generation that uses state space models whose distributions are parameterized with deep neural networks. Normalizing flows are used to define flexible distributions both in the generative model and in the inference network. To improve inference the inference networks uses samples from the prior SSM transitions borrowing ideas from importance-weighted autoencoders. I enjoyed reading this paper, as it gives many useful insights on deep state space models and more in general on probabilistic models for sequential data. Also, it introduces novel ways of parameterizing the inference network by constructing a variational approximation over the noise term rather than the state.

Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation. Papers published at the Neural Information Processing Systems Conference.